Radiology Reimagined: Advancing Clinical Practice Through AI Innovation (2025)-"Through the Radiologist's Eyes: Vetting AI Models"

"Through the Radiologist's Eyes: Vetting AI Models" – Dr. Jason Poff

Pdf Summary

This presentation by Dr. Jason A. Poff, Director of Clinical AI at Radiology Partners, outlines a radiologist-centered approach to evaluating AI imaging models before clinical deployment. Dr. Poff emphasizes the importance of a five-step validation process designed to assess the true clinical value of AI tools from the viewpoint of practicing radiologists.<br /><br />The five key steps include: (1) examining performance statistics such as sensitivity, specificity, and positive predictive value (PPV), with special attention to how disease prevalence affects PPV; (2) assessing whether AI enhances detection rates beyond radiologist capabilities; (3) reviewing illustrative "wow" cases that demonstrate AI’s clinical contributions; (4) identifying AI pitfalls, including false positives, false negatives, and challenges due to scanner variability, diverse acquisition protocols, and real-world case prevalence differing from regulatory datasets; and (5) summarizing findings to make informed deployment decisions.<br /><br />Dr. Poff points out that AI performance reported in FDA clearances often differs from real-world validation owing to differences in prevalence and clinical settings, making local validation critical. AI models may underperform in practice relative to enriched regulatory datasets due to factors like patient mix and imaging protocols. Despite imperfections, AI can aid radiologists in detecting subtle or unexpected pathology, overcoming satisfaction-of-search errors, and improving diagnostic confidence, particularly in complex cases.<br /><br />The presentation also highlights that AI and radiologist collaboration can enhance detection and patient care, but that the balance between AI-driven improvements and potential errors must be weighed carefully. Lower-performing AI models may still be valuable where timely triage or malpractice risk reduction is crucial.<br /><br />In conclusion, clinically focused AI evaluation frameworks enable better enterprise decision-making and foster radiologist engagement through education. This careful, radiologist-informed vetting is essential to advancing safe, effective integration of AI into imaging practice. Further resources are available at RSNA.org.

Keywords

Radiologist-centered AI evaluation

Clinical validation of AI imaging

AI performance metrics in radiology

Sensitivity and specificity in AI

Positive predictive value and prevalence

AI detection enhancement

AI pitfalls in medical imaging

FDA clearance vs real-world AI performance

Radiologist and AI collaboration

AI integration in clinical radiology